{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "CPBgLUU2Sb60"
   },
   "source": [
    "# Tutorial 10: Relational datasets (with star scheme)\n",
    "\n",
    "<img src=\"../../imgs/lightautoml_logo_color.png\" alt=\"LightAutoML logo\" style=\"width:100%;\"/>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "BX-Kl7zrTa-6"
   },
   "source": [
    "Official LightAutoML github repository is [here](https://github.com/sb-ai-lab/LightAutoML)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will look at how to use LightAutoML with relational datasets."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "0-fDuUjv1D_M"
   },
   "source": [
    "### Install LightAutoML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "pvNMKLp31D_Q"
   },
   "outputs": [],
   "source": [
    "#! pip install -U lightautoml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mkwewIav1D_R"
   },
   "source": [
    "### Import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "2z04SsDQ1D_R"
   },
   "outputs": [],
   "source": [
    "# Standard python libraries\n",
    "from os.path import join as pjoin\n",
    "\n",
    "# ML and DS libraries\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# Imports from lightautoml package\n",
    "from lightautoml.automl.base import AutoML\n",
    "from lightautoml.ml_algo.boost_lgbm import BoostLGBM\n",
    "\n",
    "from lightautoml.pipelines.features.lgb_pipeline import LGBSimpleFeatures\n",
    "from lightautoml.pipelines.ml.base import MLPipeline\n",
    "from lightautoml.reader.base import DictToPandasSeqReader\n",
    "from lightautoml.tasks import Task\n",
    "\n",
    "# Import Feature Generator Transformer\n",
    "from lightautoml.pipelines.features.generator_pipeline import FeatureGeneratorPipeline"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ZvilMhuZ1D_S"
   },
   "source": [
    "### Relational data\n",
    "\n",
    "Consider data that is a set of linked tables. Usually in this case there is a separate main table containing the objects identifiers and the corresponding values ​​of the target variable, as well as possibly the values ​​of other features (so called fact table). Other tables contain additional or auxiliary information, for example, records about all customer transactions (there can be an arbitrary number for a user with a specific identifier etc), the correspondence between the values ​​of one feature and the values ​​of another (the correspondence between an employee's department and his salary, for example), etc (so called dimension tables). However the organization of the data may differ from this scheme. To apply machine learning algorithms and LightAutoML, it is necessary to create a single dataset with all the features for each of the objects. For this we need to set the correspondence between the columns of the main and auxiliary tables for the correct aggregation of features. Such tables can form different schemas. \n",
    "\n",
    "In this example, we use [Meal delivery company dataset](https://www.kaggle.com/datasets/ghoshsaptarshi/av-genpact-hack-dec2018) and will consider one of the simplest and most common schemes for organizing tables - the so-called star scheme, in which there is one main table, and there are connections only between the main and auxiliary tables by specified columns, but not between separate auxiliary tables, not sequentially, etc. At the present moment, this is the only scheme supported in LightAutoML, support for more complex schemes is in development. Note that the connection between the main and each auxiliary table is carried out by a single key, but they may differ for different tables. Also, the columns for binding must be the primary key."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider an example of data with a star scheme organization. The dataset contains data on the sale of meals in the restaurant chain, consists of three tables: the main one containing information about completed orders (`train` and `test` parts), and two auxiliary tables containing information about restaurants (`fulfilment_center_info`) and available dishes (`meal_info`). The tables and the scheme of their organization are shown in the image below.\n",
    "\n",
    "<img src=\"../../imgs/Star_scheme_tables.png\" alt=\"Star scheme tables\" style=\"width:100%;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the convenience of further use, we will save datasets and paths to them in dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "Xyvl-NBw1D_T"
   },
   "outputs": [],
   "source": [
    "data_dir = '../data/meal_delivery_company'\n",
    "\n",
    "fulfilment_center_info = pd.read_csv(pjoin(data_dir, 'fulfilment_center_info.csv'))\n",
    "meal_info = pd.read_csv(pjoin(data_dir, 'meal_info.csv'))\n",
    "df_main = pd.read_csv(pjoin(data_dir, 'relational_main.csv.zip'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>center_id</th>\n",
       "      <th>city_code</th>\n",
       "      <th>region_code</th>\n",
       "      <th>center_type</th>\n",
       "      <th>op_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11</td>\n",
       "      <td>679</td>\n",
       "      <td>56</td>\n",
       "      <td>TYPE_A</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13</td>\n",
       "      <td>590</td>\n",
       "      <td>56</td>\n",
       "      <td>TYPE_B</td>\n",
       "      <td>6.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>124</td>\n",
       "      <td>590</td>\n",
       "      <td>56</td>\n",
       "      <td>TYPE_C</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>648</td>\n",
       "      <td>34</td>\n",
       "      <td>TYPE_A</td>\n",
       "      <td>4.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>94</td>\n",
       "      <td>632</td>\n",
       "      <td>34</td>\n",
       "      <td>TYPE_C</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   center_id  city_code  region_code center_type  op_area\n",
       "0         11        679           56      TYPE_A      3.7\n",
       "1         13        590           56      TYPE_B      6.7\n",
       "2        124        590           56      TYPE_C      4.0\n",
       "3         66        648           34      TYPE_A      4.1\n",
       "4         94        632           34      TYPE_C      3.6"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fulfilment_center_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>meal_id</th>\n",
       "      <th>category</th>\n",
       "      <th>cuisine</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1885</td>\n",
       "      <td>Beverages</td>\n",
       "      <td>Thai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1993</td>\n",
       "      <td>Beverages</td>\n",
       "      <td>Thai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2539</td>\n",
       "      <td>Beverages</td>\n",
       "      <td>Thai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1248</td>\n",
       "      <td>Beverages</td>\n",
       "      <td>Indian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2631</td>\n",
       "      <td>Beverages</td>\n",
       "      <td>Indian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   meal_id   category cuisine\n",
       "0     1885  Beverages    Thai\n",
       "1     1993  Beverages    Thai\n",
       "2     2539  Beverages    Thai\n",
       "3     1248  Beverages  Indian\n",
       "4     2631  Beverages  Indian"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meal_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>week</th>\n",
       "      <th>center_id</th>\n",
       "      <th>meal_id</th>\n",
       "      <th>checkout_price</th>\n",
       "      <th>base_price</th>\n",
       "      <th>emailer_for_promotion</th>\n",
       "      <th>homepage_featured</th>\n",
       "      <th>num_orders</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1476796</td>\n",
       "      <td>135</td>\n",
       "      <td>43</td>\n",
       "      <td>1770</td>\n",
       "      <td>486.03</td>\n",
       "      <td>486.03</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1168999</td>\n",
       "      <td>65</td>\n",
       "      <td>23</td>\n",
       "      <td>2760</td>\n",
       "      <td>241.53</td>\n",
       "      <td>241.53</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1190875</td>\n",
       "      <td>105</td>\n",
       "      <td>75</td>\n",
       "      <td>2444</td>\n",
       "      <td>709.13</td>\n",
       "      <td>708.13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1375454</td>\n",
       "      <td>68</td>\n",
       "      <td>10</td>\n",
       "      <td>2760</td>\n",
       "      <td>222.13</td>\n",
       "      <td>224.13</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>634</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1397113</td>\n",
       "      <td>33</td>\n",
       "      <td>36</td>\n",
       "      <td>1438</td>\n",
       "      <td>256.08</td>\n",
       "      <td>243.50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>122</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  week  center_id  meal_id  checkout_price  base_price  \\\n",
       "0  1476796   135         43     1770          486.03      486.03   \n",
       "1  1168999    65         23     2760          241.53      241.53   \n",
       "2  1190875   105         75     2444          709.13      708.13   \n",
       "3  1375454    68         10     2760          222.13      224.13   \n",
       "4  1397113    33         36     1438          256.08      243.50   \n",
       "\n",
       "   emailer_for_promotion  homepage_featured  num_orders  \n",
       "0                      0                  0          40  \n",
       "1                      0                  0          68  \n",
       "2                      0                  0          80  \n",
       "3                      0                  1         634  \n",
       "4                      0                  1         122  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_main.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45655, 9)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_main.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "O_9NMr7v1D_V"
   },
   "source": [
    "### Create sequential star scheme dictionary\n",
    "\n",
    "For further use of LightAutoML, you need to specify the data schema. It is necessary to specify secondary tables in the dictionary as the key to which the dictionary of the remaining parameters corresponds. The following parameters are specified in this dictionary:\n",
    "\n",
    "- `'case'` -  the type of column that plays the role of a key for binding. If `'ids'`, then the column is treated as a set of unique identifiers (ids), and if `'next_values'`, then it is treated as a set of timestamps.\n",
    "\n",
    "- `'params'` - dictionary of timestamp processing and interpretation parameters in case of linking by `'next_values'` type column. In case of `'ids'` it might be set empty.\n",
    "\n",
    "- `'scheme'` -  dictionary describing the scheme of relationship between the main and secondary table. Consists of the next keys: \n",
    "  - `'to'` - the name of the table, the relationship with which is being considered (in case of star scheme, the name `'plain'` should be specified here)\n",
    "  - `'from_id'` - the name of column for link in secondary table (from which the link exists); \n",
    "  - `'to_id'` - the name of column for link in main table (to which the link exists).\n",
    "\n",
    "In our example, columns for linkage are IDs. Now we set a dictionary of parameters for communication taking into account the table schema:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "5H45rr9U1D_V"
   },
   "outputs": [],
   "source": [
    "seq_params = {\n",
    "   'fulfilment_center_info': {\n",
    "      'case': 'ids',\n",
    "      'params': {},\n",
    "      'scheme': {'to': 'plain', 'from_id': 'center_id', 'to_id': 'center_id'},\n",
    "   },\n",
    "   'meal_info':{\n",
    "      'case': 'ids',\n",
    "      'params': {},\n",
    "      'scheme': {'to': 'plain', 'from_id': 'meal_id', 'to_id': 'meal_id'},\n",
    "   },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MzsUHXMM1D_V"
   },
   "source": [
    "Create a dict with second-level tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "eji2Dqh31D_W"
   },
   "outputs": [],
   "source": [
    "seq_data = {\n",
    "       'fulfilment_center_info': fulfilment_center_info,\n",
    "       'meal_info': meal_info\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Ml4JPEb1D_W"
   },
   "source": [
    "Define train and test data samples. They must be specified in the form of a dictionary, where the main dataset is specified by the `'plain'` key, and the dictionary with secondary tables is specified by the `'seq'` key (like the `seq_data` dictionary). Note that train and test data differ only in plain data, and train plain data must contain a column with the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "YG-qbhny1D_W"
   },
   "outputs": [],
   "source": [
    "train, test = train_test_split(df_main.sort_values(by='week', ascending=True), shuffle=False, test_size=0.2)\n",
    "\n",
    "train = {\n",
    "    'plain': train,\n",
    "    'seq': seq_data\n",
    "}\n",
    "\n",
    "test = {\n",
    "    'plain': test,\n",
    "    'seq': seq_data\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OVnjp0b91D_X"
   },
   "source": [
    "### Create Task snd Sequential Reader for the star scheme data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To work with linked tables in LightAutoML, it is not possible to use tabular presets like `TabularAutoML`, so we have to set all the pipeline manually. You can see more details about creating custom pipelines in [this tutorial](https://github.com/sb-ai-lab/LightAutoML/blob/master/examples/tutorials/Tutorial_6_custom_pipeline.ipynb).\n",
    "\n",
    "First we will set task and roles for our objective. Than it is necessary to create `DictToPandasSeqReader` to process data in form of relational tables. It requires setting the task and sequential data parameters dict as arguments (more details about this reader you can see [here](https://github.com/sb-ai-lab/LightAutoML/blob/master/lightautoml/reader/base.py#L651)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "VwtC0Glv1D_X"
   },
   "outputs": [],
   "source": [
    "task = Task('reg', metric='mae')\n",
    "roles={'target': 'num_orders'}\n",
    "reader = DictToPandasSeqReader(task=task, seq_params=seq_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nd6UmdxP1D_Y"
   },
   "source": [
    "### Create Feature Generator Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CgNWcG9p1D_Y"
   },
   "source": [
    "In addition to aggregating data from all related tables into one, LightAutoML has the ability to perform additional feature generation by using `FeatureGeneratorPipeline`. Features can be generated using various aggregations (taking the average, median, counting unique values, etc.), extracting date features (year, day, difference between dates, weekend or weekday, etc.), different transformations, as well as using so-called interesting values, that is, constructing features by objects with a certain value of a set of categorical features (conditional feature generation, like \"where\" clause). For aggregation and transformation LightAutoML uses according primitives from FeatureTools, detailed info is available [here](https://docs.featuretools.com/en/v0.16.0/automated_feature_engineering/primitives.html).\n",
    "\n",
    "Define interesting values parameters for feature generation in corresponding tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "QbaUFn481D_Y"
   },
   "outputs": [],
   "source": [
    "interesting_values = {\n",
    "    'fulfilment_center_info': {'center_type': ['TYPE_A', 'TYPE_C'], 'city_code': [647, 456, 703]},\n",
    "    'meal_info': {'category': ['Extras', 'Seafood'], 'cuisine': ['Continental', 'Thai']}\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, in our example we want to generate features by orders where `'center_type'` feature was equal to `'TYPE_A'` or `'TYPE_C'`, and `'city_code'` feature was equal to `647`, `456` or `703`, and similarly for features of ordered meal from `meal_info` table. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "pMCZro_M1D_Z"
   },
   "source": [
    "Params of feature generator:\n",
    "- seq_params: secondary tables or sequence related parameters.\n",
    "- max_gener_features: maximum number of generated features.\n",
    "- max_depth: maximum allowed depth of features (that is, the number of consecutively applied aggregation and transformation primitives in a superposition to obtain features).\n",
    "- agg_primitives: list of aggregation primitives. By default it is \\[`\"entropy\"`, `\"count\"`, `\"mean\"`, `\"std\"`, `\"median\"`, `\"max\"`, `\"sum\"`, `\"num_unique\"`, `\"min\"`, `\"percent_true\"`\\].\n",
    "- trans_primitives: list of transform primitives. By default it is \\[`\"hour\"`, `\"month\"`, `\"weekday\"`, `\"is_weekend\"`, `\"day\"`, `\"time_since_previous\"`, `\"week\"`, `\"age\"`, `\"time_since\"`\\].\n",
    "- interesting_values: categorical values if the form of {'table_name': {'column': [values]}} for feature generation in corresponding slices (like the `interesting_values` dictionary above).\n",
    "- generate_interesting_values: whether generate feature in slices of unique categories or not.\n",
    "- per_top_categories: percent of most frequent categories for feature generation in corresponding slices. If number of unique values is less than 10, then the all values are be used. \n",
    "- sample_size: size of data to make generated feature selection on it.\n",
    "- n_jobs: number of processes to run in parallel\n",
    "\n",
    "More details about `FeatureGeneratorPipeline` are available FeatureGeneratorPipeline class in lightautoml/pipelines/features/generator_pipeline.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "--o_H81g1D_a"
   },
   "outputs": [],
   "source": [
    "generator = FeatureGeneratorPipeline(\n",
    "    seq_params,\n",
    "    max_gener_features=500,\n",
    "    interesting_values = interesting_values,\n",
    "    generate_interesting_values = True,\n",
    "    per_top_categories = 25,\n",
    "    sample_size = None,\n",
    "    n_jobs = 16\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZX1epfUW1D_a"
   },
   "source": [
    "### Create one-level ML pipeline for AutoML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will compose the entire pipeline. We will add the basic simplest transformations to the pipeline of feature generation (encoding categorical features, converting date features to appropriate format, defining numeric types, defining roles). The set of algorithms will consist only of LightGBM gradient boosting, and no pre-selection or post-selection of features will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "rMj4jtC81D_a"
   },
   "outputs": [],
   "source": [
    "simpletransf = LGBSimpleFeatures()\n",
    "feats = generator.append(simpletransf)\n",
    "\n",
    "model = BoostLGBM()\n",
    "\n",
    "pipeline_lvl1 = MLPipeline([model], pre_selection=None, features_pipeline=feats,post_selection=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize `AutoML` instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "XukxiL-01D_b"
   },
   "outputs": [],
   "source": [
    "automl = AutoML(reader, [[pipeline_lvl1],], skip_conn=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GqErdXgS1D_b"
   },
   "source": [
    "### Train AutoML on loaded data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train our model on train data and look at the logs of training. For more detailed info we will set verbosity level to 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "tccyQs8C1D_b",
    "outputId": "783cbd8a-8017-46d6-ef97-8671bcfa4ca5",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17:03:13] Feats was rejected during automatic roles guess: []\n",
      "[17:03:13] Layer \u001b[1m1\u001b[0m train process start. Time left 9999999997.76 secs\n",
      "[17:03:13] This selector only for holdout training. fit_on_holout argument added just to be compatible\n",
      "[17:03:13] Copying TaskTimer may affect the parent PipelineTimer, so copy will create new unlimited TaskTimer\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rinchin/lama_gitlab/LightAutoML/.venv/lib/python3.8/site-packages/featuretools/synthesis/dfs.py:321: UnusedPrimitiveWarning: Some specified primitives were not used during DFS:\n",
      "  trans_primitives: ['age', 'day', 'hour', 'is_weekend', 'month', 'time_since', 'time_since_previous', 'week', 'weekday']\n",
      "  agg_primitives: ['percent_true']\n",
      "  where_primitives: ['entropy', 'num_unique', 'percent_true']\n",
      "This may be caused by a using a value of max_depth that is too small, not setting interesting values, or it may indicate no compatible columns for the primitive were found in the data. If the DFS call contained multiple instances of a primitive in the list above, none of them were used.\n",
      "  warnings.warn(warning_msg, UnusedPrimitiveWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EntitySet scattered to 16 workers in 4 seconds\n",
      "[17:03:22] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:23] [100]\tvalid's l1: 208.397\n",
      "[17:03:23] [200]\tvalid's l1: 208.432\n",
      "[17:03:24] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.028\n",
      "[17:03:24] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:24] Started iteration 0, chunk = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)']\n",
      "[17:03:24] Features in SCI = ['ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)']\n",
      "[17:03:24] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:24] [100]\tvalid's l1: 208.366\n",
      "[17:03:25] [200]\tvalid's l1: 208.36\n",
      "[17:03:25] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.044\n",
      "[17:03:25] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:25] Update best score from None to -208.04352023579347\n",
      "[17:03:25] Started iteration 1, chunk = ['ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)']\n",
      "[17:03:25] Features in SCI = ['ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_C)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)']\n",
      "[17:03:25] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:26] [100]\tvalid's l1: 208.434\n",
      "[17:03:26] [200]\tvalid's l1: 208.345\n",
      "[17:03:27] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.067\n",
      "[17:03:27] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:27] Started iteration 2, chunk = ['ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)']\n",
      "[17:03:27] Features in SCI = ['ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)']\n",
      "[17:03:27] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:27] [100]\tvalid's l1: 208.433\n",
      "[17:03:28] [200]\tvalid's l1: 208.34\n",
      "[17:03:28] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.057\n",
      "[17:03:28] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:28] Started iteration 3, chunk = ['ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)']\n",
      "[17:03:28] Features in SCI = ['ft__plain_center_id.COUNT(fulfilment_center_info WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)']\n",
      "[17:03:28] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:29] [100]\tvalid's l1: 208.432\n",
      "[17:03:29] [200]\tvalid's l1: 208.338\n",
      "[17:03:30] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.095\n",
      "[17:03:30] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:30] Started iteration 4, chunk = ['ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code)', 'ft__plain_meal_id.COUNT(meal_info)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_B)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code)', 'ft__plain_meal_id.COUNT(meal_info)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_B)']\n",
      "[17:03:30] Features in SCI = ['ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_meal_id.COUNT(meal_info)']\n",
      "[17:03:30] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:30] [100]\tvalid's l1: 208.386\n",
      "[17:03:31] [200]\tvalid's l1: 208.392\n",
      "[17:03:31] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.077\n",
      "[17:03:31] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:31] Started iteration 5, chunk = ['ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.COUNT(fulfilment_center_info)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_meal_id.ENTROPY(meal_info.cuisine)', 'ft__plain_center_id.ENTROPY(fulfilment_center_info.center_type)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.COUNT(fulfilment_center_info)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_meal_id.ENTROPY(meal_info.cuisine)', 'ft__plain_center_id.ENTROPY(fulfilment_center_info.center_type)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)']\n",
      "[17:03:31] Features in SCI = ['ft__plain_center_id.COUNT(fulfilment_center_info)', 'ft__plain_center_id.ENTROPY(fulfilment_center_info.center_type)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code)', 'ft__plain_center_id.SUM(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.SUM(fulfilment_center_info.region_code)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_meal_id.ENTROPY(meal_info.cuisine)']\n",
      "[17:03:31] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:32] [100]\tvalid's l1: 208.447\n",
      "[17:03:32] [200]\tvalid's l1: 208.39\n",
      "[17:03:33] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 208.101\n",
      "[17:03:33] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:33] Started iteration 6, chunk = ['ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)']\n",
      "[17:03:33] Features in SCI = ['ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)']\n",
      "[17:03:33] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:33] [100]\tvalid's l1: 208.41\n",
      "[17:03:34] [200]\tvalid's l1: 208.413\n",
      "[17:03:34] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 207.977\n",
      "[17:03:34] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:34] Update best score from -208.04352023579347 to -207.97743742493378\n",
      "[17:03:34] Started iteration 7, chunk = ['ft__plain_meal_id.NUM_UNIQUE(meal_info.cuisine)'], feats to check = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_meal_id.NUM_UNIQUE(meal_info.cuisine)']\n",
      "[17:03:34] Features in SCI = ['ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_meal_id.NUM_UNIQUE(meal_info.cuisine)']\n",
      "[17:03:34] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:35] [100]\tvalid's l1: 208.41\n",
      "[17:03:35] [200]\tvalid's l1: 208.413\n",
      "[17:03:35] Early stopping, best iteration is:\n",
      "[125]\tvalid's l1: 207.977\n",
      "[17:03:35] \u001b[1mLightGBM\u001b[0m fitting and predicting completed\n",
      "[17:03:35] Finally selected feats = ['ft__plain_center_id.MAX(fulfilment_center_info.op_area)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)', 'ft__plain_center_id.MAX(fulfilment_center_info.city_code)', 'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)', 'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)', 'ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)', 'ft__plain_center_id.STD(fulfilment_center_info.op_area)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)', 'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)', 'ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rinchin/lama_gitlab/LightAutoML/.venv/lib/python3.8/site-packages/featuretools/entityset/entityset.py:1738: UserWarning: Woodwork typing information on new dataframe will be replaced with existing typing information from plain\n",
      "  warnings.warn(\n",
      "2023-03-30 17:03:40,001 - distributed.worker - WARNING - Could not find data: {'bytes-7832c43e218d1860da556fd287d63146': ['tcp://127.0.0.1:33721', 'tcp://127.0.0.1:43615', 'tcp://127.0.0.1:45185']} on workers: [] (who_has: {'bytes-7832c43e218d1860da556fd287d63146': ['tcp://127.0.0.1:33721', 'tcp://127.0.0.1:43615', 'tcp://127.0.0.1:45185']})\n",
      "2023-03-30 17:03:40,004 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:45367 failed to acquire keys: {'bytes-7832c43e218d1860da556fd287d63146': ('tcp://127.0.0.1:33721', 'tcp://127.0.0.1:43615', 'tcp://127.0.0.1:45185')}\n",
      "2023-03-30 17:03:40,143 - distributed.worker - WARNING - Could not find data: {'bytes-7832c43e218d1860da556fd287d63146': ['tcp://127.0.0.1:33721', 'tcp://127.0.0.1:43615', 'tcp://127.0.0.1:45185']} on workers: [] (who_has: {'EntitySet-12f0afe6b03337e360b57b2fea5ba056': ['tcp://127.0.0.1:42689', 'tcp://127.0.0.1:44001', 'tcp://127.0.0.1:43877'], 'bytes-7832c43e218d1860da556fd287d63146': ['tcp://127.0.0.1:33721', 'tcp://127.0.0.1:43615', 'tcp://127.0.0.1:45185']})\n",
      "2023-03-30 17:03:40,144 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:45425 failed to acquire keys: {'bytes-7832c43e218d1860da556fd287d63146': ('tcp://127.0.0.1:33721', 'tcp://127.0.0.1:43615', 'tcp://127.0.0.1:45185')}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EntitySet scattered to 16 workers in 4 seconds\n",
      "[17:03:42] Start fitting \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m ...\n",
      "[17:03:42] ===== Start working with \u001b[1mfold 0\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m =====\n",
      "[17:03:42] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:42] [100]\tvalid's l1: 95.8142\n",
      "[17:03:43] [200]\tvalid's l1: 91.9066\n",
      "[17:03:43] [300]\tvalid's l1: 90.8876\n",
      "[17:03:44] [400]\tvalid's l1: 90.3315\n",
      "[17:03:45] [500]\tvalid's l1: 90.0936\n",
      "[17:03:45] [600]\tvalid's l1: 90.0952\n",
      "[17:03:46] [700]\tvalid's l1: 89.995\n",
      "[17:03:46] Early stopping, best iteration is:\n",
      "[679]\tvalid's l1: 89.9358\n",
      "[17:03:47] ===== Start working with \u001b[1mfold 1\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m =====\n",
      "[17:03:47] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:47] [100]\tvalid's l1: 93.3286\n",
      "[17:03:48] [200]\tvalid's l1: 89.2285\n",
      "[17:03:48] [300]\tvalid's l1: 88.3445\n",
      "[17:03:49] [400]\tvalid's l1: 88.2788\n",
      "[17:03:50] [500]\tvalid's l1: 88.2329\n",
      "[17:03:50] Early stopping, best iteration is:\n",
      "[442]\tvalid's l1: 88.1347\n",
      "[17:03:50] ===== Start working with \u001b[1mfold 2\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m =====\n",
      "[17:03:50] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:51] [100]\tvalid's l1: 95.2671\n",
      "[17:03:51] [200]\tvalid's l1: 91.0065\n",
      "[17:03:52] [300]\tvalid's l1: 90.3078\n",
      "[17:03:52] [400]\tvalid's l1: 89.9448\n",
      "[17:03:53] [500]\tvalid's l1: 89.8401\n",
      "[17:03:53] Early stopping, best iteration is:\n",
      "[484]\tvalid's l1: 89.7472\n",
      "[17:03:54] ===== Start working with \u001b[1mfold 3\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m =====\n",
      "[17:03:54] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:54] [100]\tvalid's l1: 92.7871\n",
      "[17:03:55] [200]\tvalid's l1: 88.5443\n",
      "[17:03:55] [300]\tvalid's l1: 87.5072\n",
      "[17:03:56] [400]\tvalid's l1: 87.3209\n",
      "[17:03:57] [500]\tvalid's l1: 87.0954\n",
      "[17:03:57] [600]\tvalid's l1: 87.1619\n",
      "[17:03:57] Early stopping, best iteration is:\n",
      "[509]\tvalid's l1: 87.0716\n",
      "[17:03:58] ===== Start working with \u001b[1mfold 4\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m =====\n",
      "[17:03:58] Training until validation scores don't improve for 100 rounds\n",
      "[17:03:58] [100]\tvalid's l1: 92.7026\n",
      "[17:03:59] [200]\tvalid's l1: 88.2765\n",
      "[17:03:59] [300]\tvalid's l1: 87.0135\n",
      "[17:04:00] [400]\tvalid's l1: 86.527\n",
      "[17:04:00] Early stopping, best iteration is:\n",
      "[396]\tvalid's l1: 86.4735\n",
      "[17:04:01] Fitting \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m finished. score = \u001b[1m-88.27262874082102\u001b[0m\n",
      "[17:04:01] \u001b[1mLvl_0_Pipe_0_Mod_0_LightGBM\u001b[0m fitting and predicting completed\n",
      "[17:04:01] Time left 9999999950.01 secs\n",
      "\n",
      "[17:04:01] \u001b[1mLayer 1 training completed.\u001b[0m\n",
      "\n",
      "CPU times: user 2min 13s, sys: 5.83 s, total: 2min 19s\n",
      "Wall time: 50 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "train_pred = automl.fit_predict(train, roles=roles, verbose=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the **\"Finally selected feats\"** line, we can see the features generated by `FeatureGenerationPipeline` and selected using LightGBM, obtained using aggregations, tarnsformations and interesting values. For example, `'ft__plain_center_id.MEDIAN(fulfilment_center_info.region_code WHERE center_type = TYPE_A)'` feature is median over `'region_code'` column in `fulfilment_center_info` table (which linked with `'plain'` dataset by `'center_id'` key) where `'center_type'` value equals `'TYPE_A'`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PHhHBkEC1D_c"
   },
   "source": [
    "### Analyze fitted model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the generated features and their importances (received from LightGBM) which we get as a result of training the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "dgLqqsZV1D_c",
    "outputId": "8c89367d-f021-4bda-d1ae-7ef733ef2e3a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ord__checkout_price                                                                       7.618603e+09\n",
       "meal_id                                                                                   5.803957e+09\n",
       "ord__base_price                                                                           4.911443e+09\n",
       "homepage_featured                                                                         2.367179e+09\n",
       "week                                                                                      2.110229e+09\n",
       "ft__plain_center_id.MAX(fulfilment_center_info.op_area)                                   1.824585e+09\n",
       "emailer_for_promotion                                                                     1.425533e+09\n",
       "center_id                                                                                 1.280621e+09\n",
       "id                                                                                        1.044121e+09\n",
       "ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)                                9.298832e+08\n",
       "ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)      8.705599e+08\n",
       "ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)                                8.173745e+08\n",
       "ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)                                   8.160740e+08\n",
       "ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)                                 8.016899e+08\n",
       "ft__plain_center_id.MAX(fulfilment_center_info.city_code)                                 7.446240e+08\n",
       "ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)        5.920852e+08\n",
       "ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)    5.403963e+08\n",
       "ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)        5.356365e+08\n",
       "ft__plain_center_id.MEAN(fulfilment_center_info.op_area)                                  3.982109e+08\n",
       "ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)                            3.171587e+08\n",
       "ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)    2.466869e+08\n",
       "ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)      1.435612e+08\n",
       "ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_B)      0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.op_area)                                   0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_B)    0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.region_code WHERE center_type = TYPE_A)    0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_C)        0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_B)        0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.city_code)                                 0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.city_code WHERE center_type = TYPE_A)      0.000000e+00\n",
       "ft__plain_center_id.NUM_UNIQUE(fulfilment_center_info.center_type)                        0.000000e+00\n",
       "ft__plain_center_id.STD(fulfilment_center_info.op_area WHERE center_type = TYPE_A)        0.000000e+00\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_imps = model.get_features_score()\n",
    "feature_imps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite a large number of features heve non-zero importances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "8KHIcWjd1D_c",
    "outputId": "ac1546f5-87c6-4320-a2eb-3044314adb1f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['ord__checkout_price', 'meal_id', 'ord__base_price',\n",
       "       'homepage_featured', 'week',\n",
       "       'ft__plain_center_id.MAX(fulfilment_center_info.op_area)',\n",
       "       'emailer_for_promotion', 'center_id', 'id',\n",
       "       'ft__plain_center_id.MEDIAN(fulfilment_center_info.op_area)',\n",
       "       'ft__plain_center_id.MAX(fulfilment_center_info.city_code WHERE center_type = TYPE_A)',\n",
       "       'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Italian)',\n",
       "       'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Thai)',\n",
       "       'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Indian)',\n",
       "       'ft__plain_center_id.MAX(fulfilment_center_info.city_code)',\n",
       "       'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_B)',\n",
       "       'ft__plain_center_id.MAX(fulfilment_center_info.region_code WHERE center_type = TYPE_A)',\n",
       "       'ft__plain_center_id.MAX(fulfilment_center_info.op_area WHERE center_type = TYPE_A)',\n",
       "       'ft__plain_center_id.MEAN(fulfilment_center_info.op_area)',\n",
       "       'ft__plain_meal_id.COUNT(meal_info WHERE cuisine = Continental)',\n",
       "       'ft__plain_center_id.MIN(fulfilment_center_info.region_code WHERE center_type = TYPE_C)',\n",
       "       'ft__plain_center_id.MIN(fulfilment_center_info.city_code WHERE center_type = TYPE_A)'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_imps.index[feature_imps > 0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EntitySet scattered to 16 workers in 4 seconds\n"
     ]
    }
   ],
   "source": [
    "test_pred = automl.predict(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OOF MAE on train: 88.27262874082102\n",
      "MAE on test: 95.97085837200986\n"
     ]
    }
   ],
   "source": [
    "print(f\"OOF MAE on train: {mean_absolute_error(train['plain'][roles['target']], train_pred.data[:, 0])}\")\n",
    "print(f\"MAE on test: {mean_absolute_error(test['plain'][roles['target']], test_pred.data[:, 0])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p5WU5rV01D_f"
   },
   "source": [
    "### Additional materials\n",
    "\n",
    "- [Official LightAutoML github repo](https://github.com/sb-ai-lab/LightAutoML)\n",
    "- [LightAutoML documentation](https://lightautoml.readthedocs.io/en/latest)\n",
    "- [LightAutoML tutorials](https://github.com/sb-ai-lab/LightAutoML/tree/master/examples/tutorials)\n",
    "- LightAutoML course:\n",
    "    - [Part 1 - general overview](https://ods.ai/tracks/automl-course-part1) \n",
    "    - [Part 2 - LightAutoML specific applications](https://ods.ai/tracks/automl-course-part2)\n",
    "    - [Part 3 - LightAutoML customization](https://ods.ai/tracks/automl-course-part3)\n",
    "- [OpenDataScience AutoML benchmark leaderboard](https://ods.ai/competitions/automl-benchmark/leaderboard)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "77738023c946bcedeeb6a5c983bcfb7849325694ebd87acbc9267f45e9f4af48"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
